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ABSTRACT 

Recent  studies  have  shown  that  with  appropriate  operator  decision  support  and  with  sufficient 
automation,  inverting  the  multiple  operators  to  single -unmanned  vehicle  control  paradigm  is  possible. 
These  studies,  however,  have  generally  focused  on  homogeneous  teams  of  vehicles,  and  have  not 
completely  addressed  either  the  manifestation  of  heterogeneity  in  vehicle  teams,  or  the  effects  of 
heterogeneity  on  operator  capacity.  An  important  implication  of  heterogeneity  in  unmanned  vehicle  teams 
is  an  increase  in  the  diversity  of  possible  team  configurations  available  for  each  operator,  as  well  as  an 
increase  in  the  diversity  of  possible  attention  allocation  schemes  that  can  be  utilized  by  operators.  To  this 
end,  this  paper  introduces  a  resource  allocation  framework  that  defines  the  strategies  and  processes  that 
lead  to  alternate  team  configurations.  The  framework  also  highlights  the  sub-components  of  operator 
attention  allocation  schemes  that  can  impact  overall  performance  when  supervising  heterogeneous 
unmanned  vehicle  teams.  Subsequently,  a  discrete  event  simulation  model  is  presented  as  a  means  to 
model  a  single  operator  supervising  multiple  heterogeneous  unmanned  vehicles.  Results  from  an 
experimental  case  study  are  then  used  to  validate  the  model,  and  make  predictions  about  operator 
performance  for  various  heterogeneous  team  configurations. 

1.0  INTRODUCTION 

Increasing  use  of  automation  in  unmanned  vehicle  (UV)  systems  has  shifted  the  human  operator’s 
responsibility  from  manually  controlling  vehicles  to  managing  vehicles  at  the  supervisory  control  level.  At 
the  supervisory  control  level,  implementation  details  of  higher-level  tasking  initiated  by  the  human  is 
delegated  to  the  automation  onboard  these  vehicles  (Sheridan  1992).  The  reduced  workload  afforded  by 
supervisory  control  has  several  implications  for  unmanned  system  operations.  One  such  ramification  is  an 
increase  in  operator  idle  time,  which  can  be  used  as  a  force  multiplier  that  allows  operators  to  supervise 
multiple  vehicles  simultaneously,  hence  inverting  the  current  many-to-one  ratio  of  operators  to  vehicles. 
Inverting  the  operator  to  vehicle  ratio  can  also  be  used  to  reduce  manning  in  situations  where  the  number 
of  vehicles  needed  to  accomplish  missions  exceeds  that  of  available  operators,  which  is  currently  a 
significant  problem  in  the  Predator  community. 

An  increasing  body  of  literature  has  examined  the  capacity  of  single  operators  to  supervise  multiple  UVs 
(Ruff,  Narayanan,  and  Draper  2002;  Olsen  and  Wood  2003;  Cummings  et  al.  2007).  This  research  has 
mainly  focused  on  the  supervision  of  a  homogeneous  set  of  UVs.  However,  as  unmanned  vehicle  system 
mission  goals  become  increasingly  demanding,  the  composition  of  UV  teams  is  likely  to  involve  vehicles 
of  varying  capabilities.  For  example,  the  military  has  proposed  future  operational  concepts  such  as 
Network  Centric  Warfare  (Alberts,  Garstka,  and  Stein  1999)  and  the  Future  Combat  System  (FCS) 
(Feickert  2005)  that  require  interoperability  among  UVs  of  varying  attributes.  In  addition  to  heterogeneity 
across  vehicle  types,  even  a  single  UV  can  have  multiple  payloads.  Thus,  multiple  mission  objectives  can 
drive  heterogeneity  in  a  system,  which  will  ultimately  lead  to  heterogeneity  in  operator  tasks. 

These  multiple  dimensions  of  heterogeneity  introduce  a  number  of  problems  in  applying  previous  models 
of  homogeneous  UVs  to  the  heterogeneous  case.  The  different  vehicle  types  that  the  team  could  be 
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composed  of  and  the  different  tasks  that  those  vehicles  could  be  assigned  present  a  complex  and 
mathematically  intractable  problem.  Moreover,  the  method  by  which  operators  allocate  their  attention  to 
the  heterogeneous  vehicles  and/or  tasks  is  likely  to  affect  system  performance.  Finally,  capturing  the 
various  operator  management  strategies  and  their  effect  on  system  performance  is  another  important 
variable  that  must  be  considered. 

This  paper  addresses  these  problems  by  introducing  a  framework  that  utilizes  resource  allocation  to 
describe  the  process  of  heterogeneous  UV  team  creation,  as  well  as  the  attention  allocation  strategies  that 
define  the  operator’s  interaction  with  the  UV  team.  The  paper  then  presents  a  discrete  event  simulation 
model  that  incorporates  the  framework’s  considerations.  An  experimental  case  study  is  then  used  to 
validate  the  ability  of  the  model  to  predict  the  effects  of  changes  in  operator  strategies  and  system  design. 

2.0  BACKGROUND 

Previous  research  that  examined  the  capacity  of  operators  supervising  multiple  homogeneous  robots  by 
Olsen  and  Goodrich  (2003)  introduced  several  temporal-based  metrics  for  describing  operator  interaction 
with  unmanned  vehicles.  Neglect  time  (NT)  was  defined  as  the  expected  amount  of  time  that  a  robot 
(which  is  representative  of  any  UV)  can  be  ignored  before  its  performance  drops  below  some  acceptable 
threshold.  Interaction  time  (IT)  was  defined  as  the  average  time  it  takes  for  a  human  to  interact  with  the 
robot  to  ensure  that  it  is  still  working  toward  mission  accomplishment.  In  the  single  robot  example  (Figure 
1(a)),  the  operator  interacts  with  the  robot  for  length  of  time  IT  and  then  ignores  it  for  length  of  time  NT, 
and  then  repeats  this  process  after  time  NT  by  interacting  with  the  robot  once  again.  In  the  multiple  robot 
case,  an  operator  would  interact  with  one  robot  at  a  time  while  neglecting  all  other  robots  (Figure  1(b)). 

One  drawback  to  this  earlier  work  is  the  lack  of  accounting  for  human  interaction  delays  and  decision 
making  inefficiencies.  An  additional  critical  variable  needed  when  modeling  human  control  of  multiple 
vehicles  is  the  concept  of  Wait  Times  (WT).  Although  it  is  possible  for  human  beings  to  multi-task, 
humans  act  as  serial  processors  in  that  they  can  only  solve  a  single  complex  task  at  a  time  (Welford  1952; 
Broadbent  1958).  While  operators  can  rapidly  switch  between  cognitive  tasks,  any  sequence  of  tasks 
requiring  complex  cognition  will  form  a  queue  and  consequently,  wait  times  will  build  (Cummings  and 
Mitchell  in  press;  Cummings  et  al.  2007).  Wait  times  can  occur  when  1)  a  vehicle  is  neglected  while  the 
operator  is  busy  interacting  with  another  vehicle,  or  2)  when  an  operator  requires  re -orientation  time  while 
switching  between  vehicles,  or  3)  when  a  vehicle  is  neglected  due  to  lack  of  operator  situation  awareness. 

IT+NT  IT+NT 

Segment  Segment  of  original  vehicle 


Figure  1.  The  relationship  of  NT  and  IT  for  (a)  a  single  vehicle,  and  (b)  multiple  vehicles. 

In  order  to  model  wait  times,  Cumming  et  al.  (2007)  proposed  a  queuing  model  of  the  human  operator 
servicing  multiple  homogeneous  UVs.  In  the  single-server  queuing  network,  the  events  that  arrive  are 
vehicles  that  require  intervention  to  bring  them  above  some  performance  threshold.  This  queuing  model 
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forms  the  basis  of  the  discrete  event  simulation  described  in  subsequent  sections  that  allowed  us  to  model 
a  single  operator  controlling  multiple  heterogeneous  vehicles. 

3.0  HETEROGENEITY  FRAMEWORK 

In  order  to  develop  better  estimates  of  both  human  capacity  for  heterogeneous  UV  teams  as  well  as  the 
impact  of  varying  mission  tasks  and  vehicles  on  operator  performance,  we  first  created  a  framework  that 
captures  the  processes  by  which  UV  teams  are  created  and  assigned  to  human  operators. 

Based  on  the  idea  of  resource  allocation,  this  framework  is  presented  in  Figure  2.  The  framework 
incorporates  the  allocation  of  three  hierarchical  resources:  vehicles,  human  operators,  and  human 
attention.  The  first  two  resources,  vehicles  and  human  operators,  are  tangible  physical  assets  that  are 
allocated  during  mission  planning,  and  it  is  through  the  allocation  of  these  assets  that  vehicle  teams  are 
defined  and  assigned  to  operators.  The  third  resource,  operator  attention,  is  an  intangible  asset  whose 
allocation  strategy  defines  the  interaction  of  the  operator  with  the  team  of  UVs.  This  framework  is  not 
meant  to  be  a  detailed  description  of  every  aspect  of  UV  assignment,  but  is  instead  meant  to  highlight  the 
role  of  resource  allocation  strategies  in  influencing  the  effectiveness  of  human -vehicle/task  interaction. 
Starting  from  the  top  left  of  Figure  2,  a  vehicle  allocation  strategy  is  depicted  as  the  method  by  which 
vehicles,  based  on  their  capabilities  (which  includes  payloads,  vehicle  specifications,  operational  domain, 
and  levels  of  automation)  are  assigned  mission-based  tasks  that  collectively  satisfy  mission  objectives. 
The  objective  of  the  vehicle  allocation  strategy  is  to  break  down  the  mission  objectives  into  tasks  that  can 
be  allocated  to  the  different  vehicles.  The  choice  of  vehicle  allocation  strategies  depends  on  the  vehicle 
capabilities,  the  mission  objectives,  and  the  timing/control  constraints  imposed  by  the  mission 
specification. 

Next,  a  personnel  allocation  strategy  is  utilized  in  order  to  allocate  an  operator  unit  (at  the  organizational 
and  individual  level)  to  one  or  more  mission  task(s).  The  choice  of  personnel  allocation  strategies  depends 
on  the  capabilities  of  the  operators,  as  well  as  the  interfaces  available  to  them.  Together,  a  vehicle 
allocation  strategy  and  a  personnel  allocation  strategy  identify  the  particular  vehicles  and  mission  tasks 
that  will  be  the  responsibility  of  each  operator  unit.  These  initial  two  steps  in  the  framework  proposed  in 
the  preceding  discussion  are  not  the  only  possible  format.  It  is  possible  for  example  to  assign  vehicles  to 
personnel  instead  of  assigning  them  to  mission  tasks.  The  main  theme,  however,  across  any  allocation 
strategy  combination  is  that  vehicles,  tasks,  and  personnel  need  to  be  assigned  in  order  to  define  the 
vehicle/task  team  that  each  operator  unit  will  be  supervising. 

The  third  and  final  strategy  in  the  framework,  the  human-attention  allocation  strategy,  is  a  function  of  the 
level  at  which  the  operator  interacts  with  each  vehicle/task,  as  well  as  the  order  by  which  the  different 
vehicles/tasks  are  serviced.  As  depicted  in  Figure  2,  operator  resource  allocation  strategies  are  dependent 
on  the  operator-task  assignment,  the  importance  of  the  mission  tasks,  and  the  urgency  of  the  mission  tasks. 

The  significance  of  the  overarching  mission  description  block  on  the  left  hand  side  of  Figure  2  represents 
the  constraints  imposed  on  the  different  strategies  throughout  the  mission  planning  and  re-planning 
phases.  An  example  of  this  is  a  time  on  target  (TOT)  constraint  that  requires  that  an  ISR  (Intelligence, 
Surveillance  and  Reconnaissance)  task  be  assigned  to  an  unmanned  aerial  vehicle  (UAV)  over  an 
unmanned  undersea  vehicle  (UUV)  due  to  the  latter  vehicle  being  unable  to  reach  the  area  of  interest  in 
time. 

The  right  hand  side  of  Figure  2  represents  the  possible  effects  of  the  allocation  strategies  for  all  three 
resources  on  overall  mission  performance.  For  example,  in  a  two-operator  mission  that  requires  the 
completion  of  two  surface  imagery  tasks  as  well  as  two  other  target  designation  tasks,  alternate  personnel 
allocation  strategies  are  possible.  Assigning  each  operator  two  of  the  same  tasks  will  likely  result  in 
different  mission  performance,  as  opposed  to  assigning  each  operator  one  of  each  task  types. 
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The  impact  of  alternate  human-attention  allocation  strategies  on  overall  mission  performance  is  the  subject 
of  interest  in  this  paper,  and  will  be  discussed  further  in  the  experiment  section.  First,  a  more  detailed 
analysis  of  human-attention  allocation  is  presented. 

3.1  Attention  Allocation  Strategy 

In  supervising  a  UV  mission,  the  operator’s  role  is  that  of  a  mission  manager  whose  task  is  to  increase 
system  performance  in  fulfilling  that  mission.  The  operator  can  interact  with  a  UV  when  either  a)  the 
automation  is  not  acting  as  expected  and  the  operator  believes  that  interaction  can  increase  performance, 
or  b)  when  an  event  occurs  that  requires  human  judgment  and  reasoning,  something  the  automation  is 
incapable  of  handling.  For  example,  in  the  case  of  a  UAV  that  is  assigned  a  laser  designation  task,  the 
operator  could  re -plan  the  vehicle  path  generated  by  automation  in  order  to  better  meet  a  time -on -target 
restriction.  The  operator’s  judgment  is  also  critical  in  deciding  whether  a  specific  target  is  the  one  that 
should  be  designated.  When  supervising  multiple  UVs,  the  operator  attention  allocation  strategy  will 
dictate  the  method  by  which  the  operator  will  supervise  the  different  vehicles. 

An  overall  attention  allocation  strategy  can  be  dissected  into  three  main  components,  which  will  be 
discussed  in  greater  detail  below:  a)  a  neglect  strategy,  b)  a  complexity  mitigation  strategy,  and  c)  a 
switching  order  strategy. 

The  neglect  strategy  affects  the  duration  of  time  for  which  the  operator  neglects  the  UVs  (i.e.,  the 
frequency  by  which  the  operator  attends  to  the  vehicles).  The  neglect  strategy  can  vary  per  vehicle,  and 
can  be  thought  of  as  the  scheme  by  which  the  operator  distributes  his/her  attention  across  the  different 
vehicles.  A  strategy  where  the  operator  services  the  vehicle  only  when  necessary  and  otherwise  allows  the 
vehicle’s  automation  to  undertake  tasks  can  be  referred  to  as  a  neglect-macro-management  strategy.  On 
the  other  hand,  a  strategy  where  the  operator  constantly  interferes  with  the  vehicle’s  automation  can  be 
referred  to  as  a  neglect-micro-management  strategy.  Other  neglect  strategies  can  exist  between  these  two 
extremes. 

The  second  component  of  human  behavior  that  influences  attention  allocation  is  the  mitigation  of  system 
complexity  through  the  use  of  cognitive  abstractions.  For  example,  operators  can  use  mental  abstractions 
to  form  vehicle  groupings  based  on  one  or  more  criteria  in  order  to  reduce  the  complexity  of  supervising 
all  the  vehicles  (Histon  et  al.  2002;  Goodrich  et  al.  2007).  Examples  of  criteria  for  grouping  vehicles 
include  the  similarity  of  vehicle  capabilities  or  task  types.  The  result  of  such  grouping  abstractions  is  to 
organize  the  vehicles  into  relevant  groups  in  order  to  simplify  the  task  of  managing  them.  For  example,  an 
operator  that  is  supervising  multiple  UVs  in  a  mission  that  includes  coastal  and  inland  surveillance  might 
elect  to  divide  the  vehicles  into  two  groups  depending  on  their  region  of  operation. 

Finally,  the  third  component  of  human-attention  allocation  is  the  order  by  which  the  different  vehicles  are 
serviced.  When  multiple  vehicles  require  operator  attention  simultaneously,  the  operator  must  select  the 
next  vehicle  to  be  serviced.  Whereas  this  selection  process  is  relatively  simple  in  the  homogeneous  case,  it 
is  much  more  involved  in  the  heterogeneous  case.  In  the  heterogeneous  case,  the  difference  in  vehicles 
capabilities  and  their  assigned  tasks  allows  for  more  diverse  selection  strategies.  For  example,  an  operator 
that  is  supervising  two  UAVs  with  heterogeneous  tasks  can  service  the  vehicles  on  a  first  come,  first  serve 
basis  (FIFO)  or  allocate  attention  to  the  UAVs  based  on  the  priority  of  their  tasks  (preemptive  priority 
queuing).  The  order  by  which  the  vehicles  are  serviced  affects  the  total  time  that  vehicles  spend  in  the 
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Figure  2.  Human-Vehicle(s)/Task(s)  Interaction  Framework. 

system,  including  the  time  they  spend  waiting  for  service  as  well  as  their  processing  time  (Mau  and  Dolan 
2006;  Sheridan  and  Tulga  1978).  In  addition  to  having  an  effect  on  wait  times,  when  a  human  operator 
switches  between  two  different  tasks,  this  is  accompanied  by  a  mental  model  switch  that  comes  at  a  time 
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(or  switch)  cost  (Goodrich,  Quigley,  and  Cosenzo  2005;  Squire,  Trafton,  and  Parasuraman  2006).  Thus 
switching  between  different  combinations  of  heterogeneous  vehicles  can  lead  to  different  switch  costs. 

4.0  OPERATOR  MODEL 

To  examine  the  impact  of  these  various  strategies  on  overall  system  performance,  a  discrete  event 
simulation  (DES)  model  of  a  heterogeneous  UV  system  was  developed.  The  DES  model  includes  a)  a 
queuing  model  of  the  human  operator  supervising  multiple  UVs,  and  b)  a  component  to  measure  overall 
system  performance,  which  will  be  discussed  in  the  next  section. 

4.1  Overview 

The  operator  model  below  was  constructed  under  the  assumption  that  the  operator  is  acting  in  a 
supervisory  control  mode  and  that  the  different  vehicles  in  the  team  are  highly  autonomous.  Therefore,  the 
vehicles  generally  only  require  operator  interaction  for  tasks  that  require  human  judgment  and  reasoning. 
The  operator  can,  however,  attend  to  a  vehicle  even  if  the  vehicle  has  not  generated  a  task  that  requires 
human  judgment  and  reasoning  by  initiating  a  re -plan  that  could  potentially  lead  to  improved  performance. 
The  operator  model  is  based  on  the  single  server  queue  with  multiple  input  streams  (Figure  3).  An  event  in 
this  queuing  model  represents  tasks  to  be  attended  to  by  the  operator.  The  operator  can  attend  to  only  one 
event  at  a  time,  and  this  is  captured  by  the  single-server  architecture  such  that  any  events  that  arrive  while 
the  operator  is  busy  will  wait  in  a  queue.  There  are  two  main  event  types  that  are  included  in  the  model;  a) 
events  generated  by  the  vehicles  due  to  the  emergence  of  tasks  that  require  human  judgment  and 
reasoning,  and  b)  events  that  the  operator  induces  by  deciding  to  re -plan  a  vehicle’s  existing  plan  (this  is 
normally  done  because  the  operator  believes  he/she  can  improve  some  metric  as  a  result  of  the  re¬ 
planning).  The  separate  input  streams  corresponding  to  each  of  the  vehicles  in  the  team  allow  for  distinct 
arrival  and  service  distributions  for  each  vehicle,  a  property  required  to  model  a  heterogeneous  UV  team. 


Event 

Stream 
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Figure  3.  Human-Vehicle(s)/Task(s)  Interaction  Framework. 


4.2  Vehicle- Generated  Events 

There  is  one  input  stream  associated  with  each  of  the  UVs  in  the  team  (1  to  n).  An  event  arrives  to  the 
system  if  the  vehicle  associated  with  the  stream  generates  a  task  that  requires  operator  judgment/reasoning 
and  the  operator  notices  the  vehicle’s  request.  A  vehicle  that  has  already  generated  an  event  must  wait  for 
operator  attention  before  it  can  continue  its  mission  and  generate  another  event.  An  event  is  therefore 
generated  for  a  particular  stream  only  if  an  event  corresponding  to  that  stream  does  not  already  exist  in  the 
system.  Thus  inter-arrival  times  for  a  stream  are  the  times  between  the  completion  of  service  for  an  event 
and  the  arrival  of  the  next  event.  The  inter-arrival  times  for  events  from  the  vehicle  associated  with  stream 
i  is  a  random  variable  (A;)  which  has  a  particular  probability  distribution.  The  distribution  of  A;  is  a 
function  of  two  main  components;  a)  the  distribution  of  the  random  variable  that  describes  the  time 
following  a  service  that  it  takes  a  vehicle  to  generate  a  task  that  requires  operator  judgment/reasoning  and, 
b)  operator  loss  of  situational  awareness  (Equation  1). 
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Ai=Xl+Z*Xi  (1) 

The  first  term  in  Equation  1 ,  X„  is  a  random  variable  that  describes  the  time  between  a  service  completion 
for  a  task  that  requires  operator  judgment  and  reasoning  and  the  generation  of  the  of  the  next  such  task. 
The  probability  distribution  describing  Xj  is  a  function  of  the  vehicle’s  level  of  automation,  the  task 
complexity,  and  the  environment  of  operation.  The  generation  of  a  task  does  not  necessarily  imply  that  the 
operator  notices  the  generated  task  as  the  rate  excludes  any  effects  due  to  loss  of  situational  awareness 
(SA). 

The  second  term  in  Equation  1,  %*X;,  represents  a  penalty  due  to  operator  loss  of  situational  awareness 
(SA),  with  y  taking  a  value  of  zero  when  the  operator  has  complete  SA  and  higher  y  indicating  degraded 
SA.  SA  is  defined  as  the  combination  of  perception  of  elements  in  the  environment,  the  comprehension  of 
their  meaning,  and  the  projection  of  the  their  status  in  the  future  (Endlsey  1995).  The  effect  of  low  SA 
(i.e.,  high  yj  is  to  create  additional  vehicle  wait  time  (WTSA)  which  increases  A,  due  to  the  operator 
taking  longer  to  notice  the  vehicle  (Cummings  et  al.  2007).  In  order  to  capture  SA,  this  model  builds  on  an 
assumption  that  SA  is  related  to  operator  utilization  (Endsley  1993).  When  operators  are  under  high  levels 
of  utilization,  it  is  assumed  that  they  are  too  busy  to  accumulate  the  information  that  is  required  to  build 
SA.  At  the  same  time,  when  operators  are  under -utilized,  it  is  presumed  that  due  to  a  low  level  of  arousal 
and  complacency,  they  could  overlook  information  from  the  environment,  which  would  also  lead  to  low 
SA. 


Figure  4.  y  curve 


In  the  DES,  the  y  variable  in  Equation  1  is  related  to  operator  utilization  through  a  parabolic  function  that 
is  concave  upwards  (Figure  4).  This  implies  that  at  both  high  and  low  operator  utilization,  y  increases 
according  to  a  quadratic  law  and  therefore  increases  A  correspondingly.  The  parabolic  relationship  is 
inspired  by  the  Yerkes  Dodson  Law  (Yerkes  and  Dodson  1908),  which  relates  operator  utilization  to 
performance.  The  y  variable  is  multiplied  by  X  in  order  to  capture  the  effect  on  A  due  to  loss  of  SA,  which 
is  a  function  of  the  rate  at  which  the  vehicle  generates  tasks  that  require  operator  intervention.  Vehicles 
that  produce  tasks  infrequently  are  serviced  less  often,  and  are  therefore  more  likely  to  be  overlooked  than 
vehicles  that  are  serviced  more  frequently. 

Also  associated  with  each  input  stream  is  a  service  rate  which  is  based  on  the  length  of  time  it  takes  the 
operator  to  interact  with  a  particular  vehicle,  corresponding  to  the  arriving  event.  The  service  rate  of 
events,  which  is  a  random  variable,  is  modeled  using  a  service  rate  distribution  (p,)  which  is  a  function  of 
two  main  components;  a)  the  distribution  of  the  rate  at  which  operators  can  service  tasks  associated  with 
vehicle  i,  and  b)  wait  times  due  to  context  switching  (Equation  2). 
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Jui  =  Yt  +  (<p  +  z)*Yj  (2) 

The  first  term  in  Equation  2,  Y;,  is  the  random  variable  that  describes  the  length  of  time  for  which  the 
operator  must  interact  with  vehicle  i  in  order  to  address  tasks  that  require  operator  judgment  and 
reasoning.  The  probability  distribution  describing  Y;  is  mainly  a  function  of  interface  and  decision  support 
quality.  The  second  term  in  Equation  2  is  a  function  of  the  context  switching  times  that  arise  when 
servicing  a  specific  vehicle,  cp  is  a  coefficient  for  calculating  the  time  penalty  due  to  switching  between 
vehicles  with  heterogeneous  capabilities,  and  r  is  a  coefficient  for  calculating  the  time  penalty  due  to 
switching  between  vehicles  with  heterogeneous  tasks.  The  switch  cost  is  not  limited  to  switching  between 
cognitively  complex  tasks,  but  exists  even  when  humans  switch  between  cognitively  simple  ones  (Rogers 
and  Monsell  1995).  For  example,  Goodrich  et  al.  (2005)  demonstrated  that  the  existence  of  context 
switching  costs  in  multi-vehicle  control  is  unavoidable,  and  that  the  amount  of  time  required  to  switch 
between  vehicles  can  be  substantial.  For  this  DES  model,  context  switching  was  accounted  for  whenever 
the  current  vehicle’s  capability  or  its  task  type  differed  from  that  of  the  last  vehicle  serviced  (these  effects 
are  captured  by  the  cp  and  x  variables  respectively).  The  effect  of  switching  times  creates  additional 
interaction  wait  times  (WTI)  which  increases  p,  due  to  the  operator  taking  longer  to  interact  with  the 
vehicle.  The  (cp  +  x)  factor  is  multiplied  by  Y  in  order  to  capture  the  fact  that  the  context  switching  time 
effect  on  p  is  a  function  of  the  vehicle’s  interaction  time. 

4.3  Operator  Neglect  Strategy  (Operator- Induced  Events) 

In  addition  to  a  stream  corresponding  to  each  of  the  vehicles,  there  is  one  additional  stream  for  operator - 
induced  events  that  the  operator  creates  by  deciding  to  re -plan  a  vehicle’s  existing  plan.  Although  vehicles 
only  generate  events  corresponding  to  tasks  that  require  operator  judgment/reasoning,  the  operator  can 
also  interrupt  a  vehicle  if  he/she  believes  they  can  improve  some  system  performance  as  a  result  of  re¬ 
planning.  The  arrival  rate  of  events  for  this  stream  is  based  on  a  distribution  of  the  rate  at  which  the 
operator  wishes  to  re -plan  (Are).  This  distribution  represents  the  operator  neglect  strategy.  For  example,  an 
aggressive  rate  of  re -planning  would  result  in  a  neglect  micro-management  strategy,  and  a  strategy 
comprised  of  less  frequent  re -plans  would  result  in  a  neglect  macro-management  strategy.  Also  associated 
with  each  operator-induced  event  is  a  service  rate  (pre)  that  describes  how  long  it  takes  the  operator  to 
complete  a  re -planning. 

5.0  PERFORMANCE  METRIC 

The  model  just  described  allows  for  the  manipulation  of  team  heterogeneity  as  well  as  the  strategies 
utilized  by  operators  in  allocating  their  attention.  In  order  to  evaluate  the  effectiveness  of  alternate 
strategies  while  supervising  teams  with  different  levels  of  heterogeneity,  a  system  performance  metric  was 
developed.  For  this  research  effort,  we  developed  a  cost  performance  model  that  captures  system 
performance,  but  also  ensures  a  heavy  penalty  when  individual  vehicle  performance  falls  below  a  certain 
threshold.  This  performance  metric  measures  different  variables  from  the  operator  model  in  order  to 
evaluate  system  performance  (Equation  3). 

Performance  =  = — - V  ( priority j  •  P  •  min(l,A(.))  (3) 

2^  priority,  , 


Each  UV’s  contribution  to  the  performance  metric  is  captured  through  one  term  in  Equation  3.  The  P, 
factor  in  each  term  represents  the  quality  of  the  operator’s  interaction  with  that  vehicle.  The  MIN(1,  A,) 
factor  represents  the  timeliness  of  the  operator’s  interaction,  and  therefore  ensures  that  the  value  added 
due  to  the  operator’s  interaction  is  weighted  by  the  punctuality  of  that  interaction.  The  metric  therefore 
reflects  both  the  timeliness  of  interaction  as  well  as  the  quality  of  interaction.  Finally,  each  term  in  the 
equation  is  weighted  by  the  priority  of  that  vehicle,  which  is  dependent  on  the  value  of  that  vehicle’s  task 
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as  a  proportion  of  the  overall  mission  objective.  These  priorities  can  also  be  predefined  during  mission 
planning  and  might  be  dictated  by  rules  of  engagement.  Mission  performance  is  therefore  most  sensitive  to 
those  assigned  tasks  with  significant  contribution  to  the  mission  objective.  A  vehicle  that  underperforms 
on  an  individual  basis  will  negatively  impact  the  performance  metric,  and  at  the  same  time,  the  metric 
measures  the  total  contribution  of  all  vehicles  which  serves  as  an  overall  mission  performance  indicator. 

One  important  factor  that  influences  both  operator  and  mission  performance  is  the  quality  of  the  human- 
computer  interface  and  associated  decision  support.  By  comparing  performance  resulting  from  alternate 
strategies,  a  conclusion  can  be  made  as  to  what  strategies  promote  the  best  performance  trends.  This  can 
encourage  system  designers  to  design  interfaces  that  amplify  these  strategies  and  mute  those  that  result  in 
less  effective  performance.  A  study  was  therefore  conducted  to  provide  as  a  preliminary  investigation  of 
the  effects  of  alternate  resource  allocation  strategies  on  system  performance. 

6.0  EXPERIMENTAL  CASE  STUDY 

To  evaluate  the  ability  of  the  model  to  accurately  predict  the  performance  characteristics  of  human-UV 
teams,  predictions  from  the  model  were  compared  to  results  from  an  experimental  study  conducted  to 
investigate  operator  performance  issues  in  the  control  of  multiple  simulated  UVs  conducting  a  simulated 
search-and-rescue  mission.  The  experimental  study  and  model  parameters  are  discussed  below. 

6.1  Software  Test-bed 

Various  aspects  of  the  software  test-bed  used  in  the  user  study  are  described  in  this  subsection,  which 
include  mission,  interface,  and  UV  behavior. 

Mission.  The  human-UV  team  (which  consisted  of  the  participant  and  multiple  simulated  UVs)  was 
assigned  the  task  of  removing  as  many  objects  as  possible  from  the  maze  in  an  8-minute  time  period.  The 
objects  were  randomly  spread  through  the  maze,  which  was  initially  unknown.  However,  as  each  UV 
moved  about  the  maze,  it  created  a  map  which  it  shared  with  the  participant  and  the  other  UVs  in  the  team. 
The  team  could  only  see  the  positions  of  six  of  the  objects  initially.  In  each  minute  of  the  session,  the 
locations  of  two  additional  objects  were  shown.  Thus,  there  were  22  possible  objects  to  collect  during  a 
session. 

An  object  was  removed  from  the  maze  (i.e.,  collected)  using  a  three-step  process.  First,  a  UV  moved  to  the 
location  of  the  object  in  the  maze  (i.e.,  target  designation,  mission  planning,  path  planning,  and  UV 
monitoring).  Second,  the  UV  “picked  up”  the  object  (i.e.,  sensor  analysis  and  scanning).  In  the  real  world, 
performing  such  an  action  might  require  the  human  operator  to  assist  in  identifying  the  object  with  video 
or  laser  data.  To  simulate  this  task,  we  asked  users  to  identify  a  city  on  a  map  of  the  mainland  United 
States  using  Google  Earth- style  software.  Third,  the  UV  carried  the  object  out  of  the  maze  via  one  of  two 
exits. 

Interface.  The  human-UV  interface  was  the  two-screen  display  shown  in  Figure  5.  On  the  left  screen,  the 
map  of  the  maze  was  displayed,  along  with  the  positions  of  the  UVs  and  (known)  objects  in  the  maze.  The 
right  screen  was  used  to  locate  the  cities.  A  participant  could  only  control  one  UV  at  a  time.  When  a  user 
desired  to  control  a  certain  UV,  s(he)  clicked  a  button  on  the  interface  corresponding  to  that  UV  (labeled 
UV1,  UV2,  etc.).  Once  the  participant  selected  the  UV,  he/she  could  direct  the  UV  by  designating  a  goal 
location  and  modifying  the  UV’s  intended  path  to  that  goal.  Designating  a  goal  for  the  UV  was  done  by 
dragging  the  goal  icon  corresponding  to  the  UV  in  question  to  the  desired  location.  Once  the  UV  received 
a  goal  command,  it  generated  and  displayed  the  path  it  intended  to  follow.  The  participant  was  allowed  to 
modify  this  path  using  the  mouse. 
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JJV  Behavior.  The  UVs’  map  of  the  maze  took  the  form  of  an  undirected  graph.  Each  edge  of  the  graph 
was  an  ordered  pair  (u,  v)  representing  a  connection  between  vertices  u  and  v  in  the  graph.  Associated  with 
each  edge  was  a  weight  indicating  the  cost  for  a  UV  to  move  along  that  edge.  Since  the  maze  was  not  fully 
known,  a  UV  had  to  choose  between  (a)  moving  along  the  shortest  path  of  the  known  maze  to  its  user- 
specified  goal  and  (b)  exploring  the  unknown  portions  of  the  maze  in  hopes  of  finding  a  shorter  path.  To 
make  this  decision,  a  UV  assumed  that  an  unmapped  edge  from  a  known  vertex  v  led  directly  to  the  goal 
position  with  a  cost  equal  to  the  Manhattan  distance  from  v  to  the  UV’s  goal,  plus  some  cost  of 
exploration  (Ce).  Each  UV  used  Dijkstra’s  algorithm  on  the  resulting  graph  to  determine  the  path  it 
intended  to  follow. 


Figure  5.  Two-screen  interface  by  which  an  operator  directed  the  UVs. 

Using  this  approach,  the  constant  Ce  determines  the  degree  to  which  the  UVs  explore  the  unknown  maze. 
Higher  values  of  Ce  result  in  less  exploration.  We  used  a  small  value  of  CEfor  a  UV  that  was  searching  for 
an  object,  and  a  higher  value  for  a  UV  that  was  carrying  an  object.  Since  users  sometimes  felt  that  the 
resulting  behavior  was  undesirable,  they  were  allowed  to  modify  a  UV’s  path  if  they  desired. 

Two  different  versions  of  UV  autonomy  were  employed  in  the  user  study.  In  the  first  condition,  called  the 
no-decision  support  (NDS)  condition,  each  UV’s  goal  destination  was  determined  completely  by  the 
human  operator.  Once  the  UV  arrived  at  its  user-defined  goal  destination,  it  did  not  move  again  until  it 
received  a  new  command  from  the  user. 

In  the  second  condition,  called  th e.  full-decision  support  (FDS)  condition,  each  UV  automatically  selected 
a  new  goal  when  it  was  left  idle.  Specifically,  a  management -by-exception  level  of  automation  was  used  in 
which  a  UV  left  idle  at  its  goal  destination,  but  not  on  an  object  in  the  maze,  waited  15  seconds  for  the 
user  to  intervene.  If  the  user  did  not  intervene,  the  UV  automatically  moved  to  the  nearest  unassigned 
object  (if  the  UV  was  searching  for  an  object)  or  the  nearest  exit  (if  the  UV  was  already  carrying  an 
object).  Additionally,  if  the  user  did  not  intervene,  UVs  automatically  chose  to  exit  the  maze  via  the 
(estimated)  nearest  exit  in  the  final  45  seconds  of  a  session.  The  FDS  condition  also  had  one  other 
additional  decision  support  tool  to  assist  the  user  in  locating  cities  on  the  map  (to  “pick  up”  objects).  This 
decision  support  tool  decreased  the  search  time  for  a  city  on  the  map  by  about  5  seconds  on  average. 

6.2  Experimental  Procedure 

The  experimental  design  was  a  2x4  factor  study.  The  decision  support  conditions  (NDS  or  FDS)  was  a 
between  subjects  factor.  UV  team  size  was  a  within  subjects  factor;  each  participant  performed  the 
search-and-rescue  mission  for  team  sizes  of  two,  four,  six,  and  eight  UVs.  The  order  in  which  the 
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participants  used  each  team  size  was  counter-balanced  throughout  the  study.  Each  participant  was  first 
randomly  assigned  to  a  decision  support  condition  (NDS  or  FDS),  and  then  was  trained  on  all  aspects  of 
the  system.  They  then  completed  three  comprehensive  practice  sessions.  Following  these  practice  sessions, 
each  participant  performed  four  test  sessions  (each  with  a  different  team  size).  Participants  were  paid  $10 
per  hour;  the  highest  scorer  also  received  a  $100  gift  certificate.  Thirty -two  participants  between  the  ages 
of  18  and  45  (mean  24.4  years  old)  participated  in  the  study,  16  in  each  condition. 

6.3  Model  Parameters 

In  order  to  compare  the  DES  model  results  to  the  human-on-the-loop  experimental  results,  vehicle¬ 
generated  and  operator-induced  distributions  in  the  case  study  were  identified.  Vehicle-generated  events 
include  both  locating  a  city  on  the  map  and  goal  assignment  in  the  case  of  the  NDS  condition,  but  only 
locating  a  city  on  the  map  in  the  FDS  condition  (since  the  UV  could  function  without  goal-assignments 
from  the  user  in  this  condition).  Thus,  five  data  sets  were  measured  from  the  experimental  data:  1)  arrival 
rate  of  vehicle-generated  events  after  the  UV  was  serviced  for  a  vehicle -generated  event;  2)  arrival  rate  of 
vehicle -generated  events  after  the  UV  was  serviced  for  an  operator-induced  event,  3)  service  times  of 
vehicle -generated  events,  4)  arrival  rate  of  operator-induced  events,  and  5)  service  times  of  operator- 
induced  events. 

The  data  sets  collected  were  then  used  to  generate  random  distributions  that  were  used  by  the  model.  In 
most  cases,  the  distribution  that  best  fit  the  data  was  the  lognormal  distribution.  This  was  expected  for 
both  service  times  and  arrival  rates  since  they  are  skewed  to  the  left,  corresponding  to  the  cases  where  an 
abnormally  long  amount  of  time  passes  between  events  or  while  the  operating  is  servicing  a  vehicle.  The 
measured  service  times  and  arrival  rates  for  vehicle-generated  and  operator  induced  tasks  are  summarized 
in  Appendix  Al. 

The  complete  model  of  the  human-UV  team  also  requires  a  performance  model.  In  the  user  study,  the 
team  scored  points  when  an  object  was  removed  from  the  maze.  In  the  NDS  condition,  this  required  two 
vehicle -generated  events  to  occur  (goal-assignment  and  locating  a  city).  Thus,  the  DES  Model  awarded  a 
point  for  the  servicing  of  every  two  vehicle -generated  events.  In  the  FDS  condition,  only  one  vehicle¬ 
generated  event  (locating  a  city)  needed  to  be  performed.  Thus,  in  this  condition,  the  DES  Model  awarded 
a  point  for  every  serviced  vehicle-generated  event. 

7.0  RESULTS  &  DISCUSSION 

In  this  section,  predictions  made  by  the  model  for  system  performance  (number  of  objects  collected)  and 
operator  utilization  (percent  operator  busy  time)  are  compared  to  observed  results  from  the  user  study. 
Operator  utilization  is  included  in  the  analysis  since  users  who  are  more  than  70%  busy  typically 
demonstrate  degraded  performance  (Schmidt  1978;  Rouse  1983;  Cummings  and  Guerlain  2007).  After 
these  predictions  are  analyzed  and  discussed,  predictions  made  by  the  model  for  heterogeneous  teams  are 
also  presented  and  discussed. 

7.1  Homogeneous  Teams 

Using  the  distributions  of  arrival  rates  and  service  times  generated  from  the  data  in  the  user  study,  10,000 
trials  were  conducted  with  the  discrete  event  simulator  to  predict  system  performance  and  operator 
utilization  for  each  condition  of  the  study. 

The  observed  system  performance  and  operator  utilization  from  the  user  study  are  compared  with  the 
model’s  estimates  in  Figures  6  and  7,  respectively.  For  the  FDS  condition,  the  model’s  predictions  for 
system  performance  are  all  within  the  95%  confidence  intervals.  Fikewise,  the  FDS  utilization  predictions 
are  all  within  the  95%  confidence  intervals  except  in  the  2-UV  case.  In  this  case,  the  model  underestimates 
operator  utilization  by  approximately  one  standard  deviation.  In  the  NDS  condition,  the  model’s 
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predictions  of  system  performance  are  within  the  95%  confidence  intervals  for  the  4-  and  6-UV 
conditions,  but  were  low  (1.6  standard  deviations  away  from  the  mean)  in  the  2-UV  condition  and  slightly 
high  in  the  8-UV  condition  (0.6  standard  deviations  from  the  mean).  Additionally,  predictions  of  operator 
utilization  are  within  the  95%  confidence  intervals  for  6-  and  8-UV  teams,  but  not  the  2-  and  4-UV  teams, 
where  predictions  are  off  by  2  and  0.8  standard  deviations  respectively. 


Figure  6.  Comparison  of  mean  performance  scores  from  predictive  model  and  case  study  for  2,  4,  6,  and  8 
vehicles  for  (a)  the  no  decision  support  case,  and  (b)  the  full  decision  support  case. 

Two  observations  about  the  accuracy  of  the  model’s  predictions  can  be  made  from  these  results.  First, 
predictions  are  more  accurate  for  larger  teams  than  small  teams.  This  trend  appears  to  be  caused  at  least  to 
some  degree  by  overly  high  penalties  associated  with  low  utilization  in  the  SA  model  (Equation  1  and 
Figure  4).  As  a  result,  the  human  is  modeled  as  not  servicing  UVs  as  often  as  necessary  in  the  2-UV 
condition.  This  leads  to  the  low  predictions  for  both  operator  utilization  and  system  performance  in  this 
condition. 

The  second  trend  observed  in  Figures  6  and  7  is  that  the  model’s  estimates  are  better  in  the  FDS  condition 
than  in  the  NDS  condition.  Again,  it  appears  that  this  trend  is  due  to  difficulties  in  modeling  human 
behavior,  as  human  behavior  is  more  difficult  to  model  than  is  automation.  Thus,  systems  that  rely  more 
on  human  behavior  (i.e.,  the  NDS  condition)  are  more  difficult  to  accurately  model  than  systems  that  rely 
more  on  automated  behavior  (i.e.,  the  FDS  condition). 

Despite  variations  in  the  accuracy  of  the  model’s  predictions,  the  predictions  do  capture  the  general  trends 
in  system  performance  and  operator  utilization  as  the  size  of  the  team  and  the  level  of  decision  support 
change.  This  result  is  important  since  it  means  that  the  model  gives  adequate  predictions  of  the  behavior 
of  different  system  architectures  in  a  cost  effective  manner.  Given  this  positive  result,  the  model  is  now 
used  to  predict  system  performance  and  operator  utilization  in  the  same  mission  for  heterogeneous  teams. 


7.2  Heterogeneous  Teams 

The  teams  considered  in  the  user  study  consisted  of  UVs  with  homogeneous  capabilities.  In  this 
subsection,  simulated  heterogeneous  UV  teams  of  sizes  2,  4,  6,  and  8  with  NDS  and  FDS  capabilities  are 
considered.  Such  teams  reflect  situations  in  which  operators  must  simultaneously  supervise  both  legacy 
UVs  with  less  autonomy  (such  as  NDS)  and  newer,  more  automated  UVs  (such  as  FDS).  The  performance 
of  each  of  these  teams  using  various  operator  strategies  is  compared  to  the  performance  of  the 
homogeneous  teams.  Different  switching  strategies  and  neglect  strategies  are  considered. 
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Switching  Strategies.  Recall  that  switching  strategies  refer  to  the  order  in  which  the  operator  attends  to 
UVs  that  need  to  be  serviced.  This  requires  that  UVs  and  tasks  be  given  specific  priorities,  and  that  the 
operator  services  the  UV  with  the  highest  priority.  For  the  heterogeneous  model  predictions,  a  first-in- 
first-out  (FIFO)  switching  scheme  as  well  as  two  different  priority  schemes  are  considered.  In  the  first 
priority  scheme  (referred  to  as  NDS-prej erred),  vehicle-generated  events  by  UVs  with  NDS  capabilities 
are  given  the  highest  priority,  followed  by  vehicle-generated  events  by  UVs  with  FDS  capabilities, 
followed  by  operator-induced  events.  For  the  second  priority  scheme  (referred  to  as  FDS-pref erred),  the 
highest  priority  is  given  to  vehicle-generated  events  from  UVs  with  FDS  capabilities,  followed  by  vehicle¬ 
generated  events  by  UVs  with  NDS  capabilities,  followed  by  operator-induced  events. 

In  Figure  8,  the  performance  scores  and  utilizations  of  the  heterogeneous  teams  for  each  of  the  three 
switching  strategies  are  compared  to  the  results  associated  with  the  two  homogeneous  teams  from  the 
previous  section.  Due  to  the  congruity  of  the  vehicles,  it  can  be  assumed  that  operators  exhibited  a  FIFO 
switching  strategy  in  the  two  homogeneous  cases.  Figure  8(a)  shows  that  all  three  heterogeneous  team 
results  have  nearly  identical  performance  scores  that  fall  in  between  the  performances  of  the  NDS  and 
FDS  homogeneous  teams.  For  teams  of  sizes  6  and  8,  operator  utilization  is  saturated  in  the 
heterogeneous  case  for  all  three  switching  strategies,  just  like  in  the  homogeneous  FDS  case  (Figure  8(b)). 
However,  performance  for  the  homogeneous  FDS  case  exceeds  the  performance  in  the  heterogeneous 
cases.  This  is  expected  since  the  heterogeneous  operators  have  a  higher  task  load  due  to  the  inclusion  of 
NDS  vehicles  in  the  teams,  which  require  more  regular  operator  attention.  In  the  homogeneous  FDS  case, 
operators  take  advantage  of  the  reduced  task  load  by  doing  extra  re -planning,  which  results  in  improved 
performance. 


4  6 

Number  of  Vehicles 


8 


Figure  8.  Comparison  of  (a)  mean  performance  scores,  and  (b)  mean  utilizations  when  using  alternate 
switching  strategies  for  teams  with  2,  4,  6,  and  8  vehicles. 


Thus  our  model  predicts  that  vehicle  team  size  and  the  level  of  decision  support  will  have  a  more  profound 
impact  on  system  performance  for  this  particular  search-and-rescue  mission  than  does  the  operator’s 
choice  of  switching  strategy.  For  this  simulated  mission,  the  system  was  relatively  robust  in  terms  of 
operator  strategy,  which  should  be  a  design  goal  in  human  supervisory  control  systems.  Because  military 
operators  receive  significant  on-the-job  training  and  experience  high  turnover  rates,  a  well-designed 
system  that  can  tolerate  wide  variability  in  strategies  is  crucial  for  future  multiple-UV  systems. 


Neglect  Strategies.  Recall  that  an  operator’s  neglect  strategy  refers  to  how  likely  she/he  is  to  re -program  a 
UV  at  any  point  in  time.  In  the  heterogeneous  case,  it  can  be  important  to  use  different  neglect  strategies 
for  different  UV  systems.  For  example,  a  highly  autonomous  UV  may  need  to  be  re -planned  at  a  different 
rate  than  a  less  autonomous  UV. 


For  the  heterogeneous  model  predictions,  three  neglect  strategies  are  analyzed.  In  the  first  strategy 
(referred  to  as  50/50),  the  operator  applies  an  equal  neglect  strategy  to  UVs  with  NDS  and  FDS 
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capabilities.  In  the  second  strategy  (called  Re-plan  NDS),  the  operator  chooses  to  re -plan  UVs  with  NDS 
capabilities,  but  not  UVs  with  FDS  capabilities.  In  the  third  strategy  (called  Re-plan  FDS ),  the  operator 
re -plans  UVs  with  FDS  capabilities,  but  not  UVs  with  NDS  capabilities. 

The  system  performance  and  operator  utilization  of  these  teams  with  the  three  operator  strategies  are 
compared  to  NDS  and  FDS  homogeneous  teams  in  Figure  9.  Figure  9(a)  shows  that  as  with  the  previous 
predictions,  the  homogeneous  team  operators  with  full  decision  support  achieved  the  highest  performance 
due  to  the  lower  task  and  mission  complexity.  In  addition,  those  heterogeneous  operators  that  focused  on 
replanning  with  no  decision  support  performed  at  the  same  degraded  level  as  those  operators  with  a 
homogeneous  team  with  no  decision  support.  Interestingly,  these  same  operators  experienced  the  lowest 
utilization,  so  they  had  spare  cognitive  resources  but  performed  at  a  degraded  level  because  they  did  not 
efficiently  allocate  their  attention.  These  results  illustrate  the  need  to  provide  decision  support,  regardless 
of  the  team  composition,  as  well  as  directed  decision  support  that  makes  it  clear  to  the  operator  what  tasks 
are  the  most  critical. 

Another  interesting  result  is  that  when  operators  re -plan  UVs  with  FDS  capabilities  (i.e.,  in  50/50  and  Re¬ 
plan  FDS),  the  performance  of  the  team  improves.  These  results  imply  that  when  transitioning  from 
legacy  non-highly  autonomous  vehicles  to  more  autonomous  vehicles,  operators  supervising  mixed  teams 
can  achieve  higher  levels  of  performance,  but  that  they  need  to  understand  when  and  how  to  override  the 
automation  when  replanning. 


Figure  9.  Comparison  of  (a)  mean  performance  scores,  and  (b)  mean  utilizations  when  using  alternate  neglect 

strategies  for  teams  with  2,  4,  6,  and  8  vehicles. 


8.0  CONCLUSIONS 

A  framework  was  presented  that  identified  the  resource  allocation  strategies  that  are  fundamental  in 
defining  the  types  of  heterogeneity  that  will  be  present  in  vehicle  teams,  as  well  as  the  different  attention 
allocation  strategies  that  are  available  to  the  human  operator  in  supervising  such  teams.  Based  on  this 
framework,  a  discrete-event  simulation  model  was  developed  to  investigate  the  effect  of  alternate  operator 
strategies  and  team  compositions  in  supervising  multiple  heterogeneous  unmanned  vehicles.  In  this  paper, 
the  model  was  used  to  predict  the  system  performance  and  operator  utilization  of  multiple  simulated 
homogeneous  UV  teams.  Comparisons  of  these  predictions  with  observed  human-in-the-loop 
experimental  results  show  that  the  model  adequately  captures  these  system  dynamics  for  these 
homogeneous  cases. 

Because  this  model  allows  for  alternative  configurations  both  in  vehicle  assignments  and  in  operator 
strategies,  it  is  useful  for  modeling  other  system  configurations.  In  this  paper,  predictions  were  made  for 
heterogeneous  teams  of  different  sizes  and  capabilities,  and  were  compared  to  the  performance  scores  and 
utilizations  obtained  in  the  user  study  for  homogeneous  teams.  The  performance  in  the  heterogeneous 
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cases  was  bounded  by  the  results  obtained  with  homogenous  teams,  demonstrating  that  this  simulated 
search  and  rescue  human-robot  system  was  relatively  robust  to  changes  in  operator  strategies,  but  that  the 
presence  and  correct  use  of  the  automated  decision  support  in  heterogeneous  UV  teams  is  critical.  While 
these  results  are  encouraging,  the  model  must  still  be  validated  for  heterogeneous  teams  in  future  work. 
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11.0  APPENDIX 


Al:  Summary  of  case  study  results  for  the  no  decision  support  trials 


#  of 

Vehicles 

Events 

Re-Planning 

Service  Time 

Arrival  Rates 

Decision 

Sunoort 

Mean 

2 

OI 

78.51  s 

7.17714s 

60.84  s 

ND 

Distribution 

2 

OI 

Lognormal 

Lognormal 

Lognormal 

ND 

Parameters 

2 

OI 

0=1.1871  4=3.6762 

0=0.83977  4=1.6274 

0=1.0759  4=3.5615 

ND 

Mean 

2 

VG 

67.25  s 

16.747  s 

ND 

Distribution 

2 

VG 

Gamma 

Exponential 

ND 

Parameters 

2 

VG 

a=0.4444  p=  151.3 

>.  0.05971 

ND 

Mean 

2 

OI 

70.08  s 

5.645  s 

25.187  s 

F 

Distribution 

2 

OI 

Lognormal 

Lognormal 

Exponential 

F 

Parameters 

2 

OI 

o=1.2099  4=3.550 

0=0.76877  4=1.4283 

>.=0,0397 

F 

Mean 

2 

VG 

86.48  s 

22.79  s 

F 

Distribution 

2 

VG 

Lognormal 

Lognormal 

F 

Parameters 

2 

VG 

0=0.5138  4=4.307 

0=0.4093  4=3.026 

F 

Mean 

4 

OI 

45.55  s 

4.9  s 

77.31  s 

ND 

Distribution 

4 

OI 

Lognormal 

Lognormal 

Lognormal 

ND 

Parameters 

4 

OI 

o=0. 9985(1  |.t=3.246 

o=0. 7005  4=1.312 

0=1.234  4=3.685 

ND 

Mean 

4 

VG 

54.26  s 

14.2037  s 

ND 

Distribution 

4 

VG 

Lognormal 

Gamma 

ND 

Parameters 

4 

VG 

o=l.l30  4=3.351 

0=0.66218  4=1.2805 

ND 

Mean 

4 

OI 

72.87  s 

4.509  s 

70.08  s 

F 

Distribution 

4 

OI 

Lognormal 

Lognormal 

Lognormal 

F 

Parameters 

4 

OI 

o=1.2244  4=3.6333 

0=0.76877  4=1.4283 

0=1.201  4=3.550 

F 

Mean 

4 

VG 

107.93  s 

19.76  s 

F 

Distribution 

4 

VG 

Lognormal 

Lognormal 

F 

Parameters 

4 

VG 

0=0.637  4=4.4525 

0=0.37596  4=2.8978 

F 

Mean 

6 

OI 

5 1 .06  s 

5.724  s 

81.71  s 

ND 

Distribution 

6 

OI 

Lognormal 

Lognormal 

Lognormal 

ND 

Parameters 

6 

OI 

0=1.0456  n=3.3778 

0=0.73413  4=1.4213 

0=1.328  4=3.6353 

ND 

Mean 

6 

VG 

58.93  s 

12.57  s 

ND 

Distribution 

6 

VG 

Lognormal 

Gamma 

ND 

Parameters 

6 

VG 

0=1.1743  |I=3.4I69 

a=0. 94565  (1=13.292 

ND 

Mean 

6 

OI 

82.83s 

4.331  s 

23.90253  s 

F 

Distribution 

6 

OI 

Lognormal 

Lognormal 

Lognormal 

F 

Parameters 

6 

OI 

0=1.2356  n=3.7949 

0=0.63637  4=1.2603 

0=1.2272  4=2.4277 

F 

Mean 

6 

VG 

103.616  s 

19.81  s 

F 

Distribution 

6 

VG 

Lognormal 

Lognormal 

F 

Parameters 

6 

VG 

o= 0.53908  n  =4.4863 

0=0.37706  4=2.9057 

F 

Mean 

8 

OI 

60.56  s 

4.558  s 

76.44  s 

ND 

Distribution 

8 

OI 

Lognormal 

Lognormal 

Gamma 

ND 

Parameters 

8 

OI 

o=1.175  4=3.3932 

o= 0.62325  4=1.3214 

cx=0.60371  (3=126.63 

ND 

Mean 

8 

VG 

67.25  s 

12.56  s 

ND 

Distribution 

8 

VG 

Gamma 

Lognormal 

ND 

Parameters 

8 

VG 

u=0.4444  (1=151.3 

0=1.0852  4=1.9596 

ND 

Mean 

8 

OI 

91.68  s 

4.1  1  1  s 

28.45  s 

F 

Distribution 

8 

OI 

Lognormal 

Lognormal 

Lognormal 

F 

Parameters 

8 

OI 

o=1.226l  4=3. 9041 

0=0.59295  4=1.2155 

0=1.3543  4=2.3909 

F 

Mean 

8 

VG 

1 13.8  s 

20.62s 

F 

Distribution 

8 

VG 

Lognormal 

Lognormal 

F 

Parameters 

8 

VG 

0=0.54382  (a=4.5835 

0=0.40095  4=2.9132 

F 

RTO-EN-SCI-208 


3-17 


UNCLASSIFIED/UNLIMITED 


UNCLASSIFIED/UNLIMITED 


The  Impact  of  Heterogeneity  on  Operator  Performance 
in  Futuristic  Unmanned  Vehicle  Systems 


3-18 


RTO-EN-SCI-208 


UNCLASSIFIED/UNLIMITED 


